Image processing apparatus and image processing system

ABSTRACT

Provided is an image processing apparatus ( 200 ), comprising: a communicating unit ( 202 ) capable of communicating with each of a plurality of image sensors configured to transmit, in respectively different packets, additional data including region information corresponding to a region set with respect to a captured image for each region and region image data indicating an image for each row corresponding to the region; and a processing unit ( 204 ) configured to process, in association with each region, the region image data acquired from each of the plurality of image sensors based on the region information included in the additional data acquired from each of the plurality of image sensors, wherein the region information includes a part of or all of identification information of the region, information indicating a position of the region, and information indicating a size of the region.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of application Ser. No.17/261,371, filed Jan. 19, 2021, which is a 371 National Stage Entry ofInternational Application No.: PCT/JP2019/031780, filed on Aug. 9, 2019,which claims the benefit of Japanese Priority Patent Application JP2018-153930 filed on Aug. 20, 2018, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus and animage processing system.

BACKGROUND ART

Techniques related to a compound-eye imaging apparatus that includes aplurality of imaging units are being developed. An example of suchtechniques is a technique described in PTL 1 below.

CITATION LIST Patent Literature

[PTL 1]

-   JP 2007-110499 A

SUMMARY Technical Problem

For example, in a case where the technique described in PTL 1 is used,when a specific object is detected from an image obtained by imaging byone of the imaging units constituting an imaging apparatus, imaging bythe other imaging units constituting the imaging apparatus is performed.However, using the technique described in PTL 1 simply enables aplurality of images to be obtained by imaging and no particularconsideration is given to associatively processing the plurality ofimages obtained by imaging.

The present disclosure proposes a novel and improved image processingapparatus and image processing system capable of associativelyprocessing images respectively obtained from a plurality of imagesensors.

Solution to Problem

The present disclosure provides an image processing apparatus,including: a communicating unit capable of communicating with each of aplurality of image sensors configured to transmit, in respectivelydifferent packets, additional data including region informationcorresponding to a region set with respect to a captured image for eachregion and region image data indicating an image for each rowcorresponding to the region; and a processing unit configured toprocess, in association with each region, the region image data acquiredfrom each of the plurality of image sensors based on the regioninformation included in the additional data acquired from each of theplurality of image sensors, wherein the region information includes apart of or all of identification information of the region, informationindicating a position of the region, and information indicating a sizeof the region.

In addition, the present disclosure provides an image processing system,including: a plurality of image sensors configured to transmit, inrespectively different packets, additional data including regioninformation corresponding to a region set with respect to a capturedimage for each region and region image data indicating an image for eachrow corresponding to the region; and an image processing apparatus,wherein the image processing apparatus includes: a communicating unitcapable of communicating with each of the plurality of image sensors;and a processing unit configured to process, in association with eachregion, the region image data acquired from each of the plurality ofimage sensors based on the region information included in the additionaldata acquired from each of the plurality of image sensors, and theregion information includes a part of or all of identificationinformation of the region, information indicating a position of theregion, and information indicating a size of the region.

Advantageous Effects of Invention

According to the present disclosure, images respectively obtained from aplurality of image sensors can be associatively processed.

It should be noted that the advantageous effect described above is notnecessarily restrictive and, in addition to the advantageous effectdescribed above or in place of the advantageous effect described above,any of the advantageous effects described in the present specificationor other advantageous effects that can be comprehended from the presentspecification may be produced.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram showing an example of a configurationof an information processing system according to the present embodiment.

FIG. 2 is an explanatory diagram showing a format of a packet defined inthe MIPI CSI-2 standard.

FIG. 3 is an explanatory diagram showing a format of a packet defined inthe MIPI CSI-2 standard.

FIG. 4 is an explanatory diagram showing an example of a signal waveformrelated to transmission of a packet in the MIPI CSI-2 standard.

FIG. 5 is an explanatory diagram showing an example of regions to be setwith respect to an image.

FIG. 6 is an explanatory diagram showing an example of data to betransmitted by a first transmission system related to a transmissionmethod according to the present embodiment.

FIG. 7 is an explanatory diagram for explaining an example of EmbeddedData to be transmitted by the first transmission system according to thepresent embodiment.

FIG. 8 is an explanatory diagram for explaining an example of regioninformation included in the Embedded Data shown in FIG. 7 .

FIG. 9 is an explanatory diagram showing another example of regions tobe set with respect to an image.

FIG. 10 is an explanatory diagram showing an example of data to betransmitted by a second transmission system related to the transmissionmethod according to the present embodiment.

FIG. 11 is a block diagram showing an example of a configuration of animage sensor according to the present embodiment.

FIG. 12 is a block diagram showing an example of a configuration of animage processing apparatus according to the present embodiment.

FIG. 13 is a block diagram showing an example of a functionalconfiguration of a communication circuit included in the imageprocessing apparatus according to the present embodiment.

FIG. 14 is a block diagram showing an example of a functionalconfiguration of an image processing circuit included in the imageprocessing apparatus according to the present embodiment.

FIG. 15 is an explanatory diagram for explaining an example ofprocessing in the information processing system according to the presentembodiment.

FIG. 16 is an explanatory diagram for explaining an example ofprocessing in the information processing system according to the presentembodiment.

FIG. 17 is an explanatory diagram for explaining an example ofprocessing in the information processing system according to the presentembodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a preferred embodiment of the present disclosure will bedescribed in detail with reference to the accompanying drawings. In thepresent specification and the drawings, components having substantiallya same functional configuration will be denoted by same reference signsand overlapping descriptions thereof will be omitted.

In addition, hereinafter, descriptions will be given in an orderdescribed below.

-   -   1. Transmission method according to present embodiment, image        processing method according to present embodiment, and image        processing system according to present embodiment    -   [1] Configuration of image processing system to which        transmission method according to present embodiment can be        applied    -   [2] Application example of image processing system according to        present embodiment    -   [3] Transmission method according to present embodiment    -   [4] Configuration example of image sensor and image processing        apparatus constituting image processing system according to        present embodiment    -   [5] Example of processing in image processing system according        to present embodiment    -   [6] Example of advantageous effect produced by using image        processing system according to present embodiment    -   2. Program according to present embodiment

Transmission Method According to Present Embodiment, Image ProcessingMethod According to Present Embodiment, and Image Processing SystemAccording to Present Embodiment

[1] Configuration of Image Processing System to which TransmissionMethod According to Present Embodiment can be Applied

First, an example of a configuration of an image processing system towhich a transmission method according to the present embodiment can beapplied will be described.

Hereinafter, a case where a communication system between apparatusesthat constitute the image processing system according to the presentembodiment is a communication system in compliance with the MIPI (MobileIndustry Processor Interface) CSI-2 (Camera Serial Interface 2) standardwill be described as an example. However, the communication systembetween apparatuses that constitute the image processing systemaccording to the present embodiment is not limited to a communicationsystem in compliance with the MIPI CSI-2 standard. For example,communication between apparatuses that constitute the image processingsystem according to the present embodiment may satisfy another standarddeveloped by the MIPI Alliance such as a communication system incompliance with the MIPI CSI-3 standard or a communication system incompliance with the MIPI DSI (Digital Serial Interface) standard. Inaddition, it is needless to say that communication systems to which thetransmission method according to the present embodiment can be appliedare not limited to communication systems related to standards developedby the MIPI Alliance.

FIG. 1 is an explanatory diagram showing an example of a configurationof an image processing system 1000 according to the present embodiment.Examples of the image processing system 1000 include a communicationapparatus such as a smartphone and a mobile body such as a drone (adevice that can be operated by remote control or a device capable ofoperating autonomously) or an automobile. Application examples of theimage processing system 1000 are not limited to the example describedabove. Other application examples of the image processing system 1000will be described later.

For example, the image processing system 1000 has image sensors 100A and100B, an image processing apparatus 200, a memory 300, and a displaydevice 400. Hereinafter, the term “image sensor 100” will be used whencollectively referring to the image sensors 100A and 100B and whenreferring to one image sensor of the image sensors 100A and 100B.

The image sensor 100 has an imaging function and a transmittingfunction, and transmits data indicating an image generated by imaging.The image processing apparatus 200 receives the data transmitted fromthe image sensor 100 and processes the received data. In other words, inthe image processing system 1000, the image sensor 100 performs a roleof a transmitting apparatus and the image processing apparatus 200performs a role of a receiving apparatus.

While FIG. 1 shows the image processing system 1000 having two imagesensors 100, the number of the image sensors 100 included in the imageprocessing system according to the present embodiment is not limited tothe example shown in FIG. 1 . For example, the image processing systemaccording to the present embodiment may have three or more image sensors100.

In addition, in the image processing system according to the presentembodiment, a plurality of the image sensors 100 may be modularized. Forexample, an image sensor module obtained by modularizing a plurality ofthe image sensors 100 is provided with a plurality of the image sensors100, a processor (not illustrated) for the image sensor module, and arecording medium that is readable by the processor. The recording mediumconstituting the image sensor module records, for example, informationrelated to an angle of view (for example, data indicating an angle ofview or the like) of the image sensor 100 that constitutes the imagesensor module. In addition, the processor that constitutes the imagesensor module transmits information related to the angle of view to theimage processing apparatus 200 via an arbitrary transmission path.

Furthermore, while FIG. 1 shows the image processing system 1000 havingone image processing apparatus 200, the number of the image processingapparatus 200 included in the image processing system according to thepresent embodiment is not limited to the example shown in FIG. 1 . Forexample, the image processing system according to the present embodimentmay have two or more image processing apparatuses 200. In the imageprocessing system having a plurality of the image processing apparatuses200, a plurality of image sensors 100 correspond to each of the imageprocessing apparatuses 200. Even in an image processing system having aplurality of the image sensors 100 and a plurality of the imageprocessing apparatuses 200, communication is performed between each ofthe image sensors 100 and the image processing apparatuses 200 in asimilar manner to the image processing system 1000 shown in FIG. 1 .

The image sensor 100 and the image processing apparatus 200 areelectrically connected by a data bus B1. The data bus B1 is atransmission path of a signal that connects the image sensor 100 and theimage processing apparatus 200 to each other. For example, dataindicating an image (hereinafter, sometimes described as “image data”)transmitted from the image sensor 100 is transmitted from the imagesensor 100 to the image processing apparatus 200 via the data bus B1.

A signal transmitted by the data bus B1 in the image processing system1000 is transmitted according to a communication system in compliancewith a prescribed standard such as the MIPI CSI-2 standard.

FIGS. 2 and 3 are explanatory diagrams showing a format of a packetdefined in the MIPI CSI-2 standard. FIG. 2 shows a format of a shortpacket defined in the MIPI CSI-2 standard, and FIG. 3 shows a format ofa long packet defined in the MIPI CSI-2 standard.

A long packet refers to a piece of data constituted by a packet header(“PH” shown in FIG. 3 ), a payload (“Payload Data” shown in FIG. 3 ),and a packet footer (“PF” shown in FIG. 3 ). A short packet refers to apiece of data having a structure similar to the packet header (“PH”shown in FIG. 3 ) as shown in FIG. 2 .

In both a short packet and a long packet, a VC (Virtual Channel) number(“VC” shown in FIGS. 2 and 3 : a VC value) is recorded in the headerportion and an arbitrary VC number may be assigned for each packet.Packets to which a same VC number is assigned are handled as packetsbelonging to a same piece of image data.

In addition, in both a short packet and a long packet, a DT (Data Type)value (“Data Type” shown in FIGS. 2 and 3 ) is recorded in the headerportion.

Therefore, in a similar manner to the VC number, packets to which a sameDT value is assigned can also be handled as packets belonging to a samepiece of image data.

An end of the packet is recorded as the number of words in Word Count inthe header portion of a long packet. An error correcting code isrecorded in ECC in the header portion of a short packet and a longpacket.

In the MIPI CSI-2 standard, a high-speed differential signal is usedduring a period in which a data signal is transmitted and a low powersignal is used during a blanking period of the data signal. In addition,the period in which the high-speed differential signal is used isreferred to as a period in an HPS (High Speed State) and the period inwhich the low power signal is used is referred to as a period in an LPS(Low Power State).

FIG. 4 is an explanatory diagram showing an example of a signal waveformrelated to transmission of a packet in the MIPI CSI-2 standard. A inFIG. 4 shows an example of transmission of a packet and B in FIG. 4shows another example of transmission of a packet. “ST”, “ET”, “PH”,“PF”, “SP”, and “PS” shown in FIG. 4 are, respectively, abbreviations ofthe following.

-   -   ST: Start of Transmission    -   ET: End of Transmission    -   PH: Packet Header    -   PF: Packet Footer    -   SP: Short Packet    -   PS: Packet Spacing

As shown in FIG. 4 , it is revealed that, between a differential signal(“LPS” shown in FIG. 4 ) in the period of the LPS and a differentialsignal (other than “LPS” shown in FIG. 4 ) in the period of the HPS,amplitudes of the differential signals differ from one another.Therefore, from the perspective of improving transmission efficiency aperiod of the LPS is included as little as possible.

For example, the image sensor 100 and the image processing apparatus 200are electrically connected to each other by a control bus B2 thatdiffers from the data bus B1. The control bus B2 is a transmission pathof other signals that connects the image sensor 100 and the imageprocessing apparatus 200 to each other.

For example, control information output from the image processingapparatus 200 is transmitted from the image processing apparatus 200 tothe image sensor 100 via the control bus B2.

For example, the control information includes information for controland a processing command. Examples of the information for controlinclude data for controlling functions in the image sensor 100 such asone or two or more pieces of data among data indicating an image size,data indicating a frame rate, and data indicating an amount of outputdelay from reception of an output command of an image to output of theimage. In addition, the control information may include identificationinformation indicating the image sensor 100. Examples of theidentification information include an arbitrary piece of data thatenables the image sensor 100 to be identified such as an ID set to theimage sensor 100.

The information transmitted from the image processing apparatus 200 tothe image sensor 100 via the control bus B2 is not limited to theexample described above. For example, the image processing apparatus 200may transmit, via the control bus B2, region designation informationthat designates a region in an image. Examples of the region designationinformation include data in an arbitrary format that enables a region tobe identified such as data indicating a position of a pixel included inthe region (for example, coordinate data in which a position of a pixelincluded in the region is expressed by coordinates).

While FIG. 1 shows an example in which the image sensor 100 and theimage processing apparatus 200 are electrically connected to each otherby the control bus B2, the image sensor 100 and the image processingapparatus 200 may not be connected by the control bus B2. For example,the image sensor 100 and the image processing apparatus 200 may transmitand receive control information and the like by wireless communicationbased on an arbitrary communication system.

In addition, while the image sensor 100A and the image sensor 100B arenot electrically connected to each other in FIG. 1 , alternatively, theimage sensor 100A and the image sensor 100B may be electricallyconnected to each other by a transmission path that enablescommunication to be performed by an arbitrary communication system. Whenthe image sensor 100A and the image sensor 100B are electricallyconnected to each other, the image sensor 100A and the image sensor 100Bcan directly communicate with each other. As an example, the imagesensor 100A and the image sensor 100B can communicate with each other byinter-processor communication that involves communicating via atransmission path between processors respectively included in the imagesensors 100A and 100B. Even when the image sensor 100A and the imagesensor 100B are not electrically connected to each other as shown inFIG. 1 , the image sensor 100A and the image sensor 100B can communicatewith each other via the image processing apparatus 200.

Hereinafter, each apparatus constituting the image processing system1000 shown in FIG. 1 will be described.

[1-1] Memory 300

The memory 300 is a recording medium included in the image processingsystem 1000. Examples of the memory 300 include a volatile memory suchas a RAM (Random Access Memory) and a nonvolatile memory such as a flashmemory.

The memory 300 operates using power supplied from an internal powersupply (not illustrated) that constitutes the image processing system1000 such as a battery or power supplied from an external power supplyof the image processing system 1000.

For example, the memory 300 stores an image output from the image sensor100. Recording of the image to the memory 300 is controlled by, forexample, the image processing apparatus 200.

[1-2] Display Device 400

The display device 400 is a display device included in the imageprocessing system 1000. Examples of the display device 400 include aliquid crystal display and an organic EL display (organicelectro-luminescence display). The display device 400 operates usingpower supplied from an internal power supply (not illustrated) thatconstitutes the image processing system 1000 such as a battery or powersupplied from an external power supply of the image processing system1000.

For example, various images and screens such as an image output from theimage sensor 100, a screen related to an application to be executed inthe image processing apparatus 200, and a screen related to an UI (userinterface) are to be displayed on a display screen of the display device400. Display of an image or the like on the display screen of thedisplay device 400 is controlled by, for example, the image processingapparatus 200.

[1-3] Image Sensor 100

The image sensor 100 has an imaging function and a transmittingfunction, and transmits data indicating an image generated by imaging.As described earlier, the image sensor 100 performs a role of atransmitting apparatus in the image processing system 1000.

Examples of the image sensor 100 include an imaging device such as adigital still camera, a digital video camera, or a stereo camera and animage sensor device of an arbitrary system that is capable of generatingan image such as an infrared sensor or a distance image sensor, and theimage sensor 100 has a function of transmitting a generated image. Animage generated in the image sensor 100 corresponds to data indicating asensing result in the image sensor 100. An example of a configuration ofthe image sensor 100 will be described later.

The image sensor 100 transmits image data (hereinafter, referred to as“region image data”) that corresponds to a region set with respect to animage by a transmission method according to the present embodiment to bedescribed later. Control related to the transmission of the region imagedata is performed by, for example, a component (to be described later)that functions as an image processing unit in the image sensor 100. Aregion set with respect to an image may be referred to as a ROI (Regionof Interest). Hereinafter, a region set with respect to an image may bereferred to as a “ROI”.

Examples of processing related to setting a region with respect to animage include arbitrary processing that enables a partial region in theimage to be identified (or arbitrary processing that enables a partialregion in the image to be cut out) such as “processing for detecting anobject from an image and setting a region including the detected object”or “processing for setting a region designated by an operation performedwith respect to an arbitrary operating device”.

The processing related to setting a region with respect to an image maybe performed by the image sensor 100 or performed by an externalapparatus such as the image processing apparatus 200. When the imagesensor 100 performs the processing related to setting a region withrespect to an image, the image sensor 100 identifies a region accordingto a result of the processing related to setting a region with respectto an image. In addition, for example, when an external apparatusperforms the processing related to setting a region with respect to animage, the image sensor 100 identifies a region based on regiondesignation information acquired from the external apparatus.

By having the image sensor 100 transmit region image data or, in otherwords, transmit data of a part of an image, a data amount related totransmission is reduced as compared to transmitting an entire image.Therefore, by having the image sensor 100 transmit region image data,for example, various advantageous effects are produced by the reductionin the data amount such as a reduction in transmission time and areduction in load related to transmission in the image processing system1000.

Alternatively, the image sensor 100 can also transmit data indicating anentire image.

When the image sensor 100 has a function of transmitting region imagedata and a function of transmitting data indicating an entire image, theimage sensor 100 can selectively switch between transmitting regionimage data and transmitting data indicating an entire image.

For example, the image sensor 100 transmits region image data ortransmits data indicating an entire image depending on an operating modebeing set. Setting of an operating mode is performed by, for example, anoperation with respect to an arbitrary operating device.

In addition, the image sensor 100 may selectively switch betweentransmitting region image data and transmitting data indicating anentire image based on region designation information acquired from anexternal apparatus. For example, when region designation information isacquired from the external apparatus, the image sensor 100 transmitsregion image data of a region corresponding to the region designationinformation, and when region designation information is not acquiredfrom the external apparatus, the image sensor 100 transmits dataindicating an entire image.

[1-4] Image Processing Apparatus 200

The image processing apparatus 200 receives the data transmitted fromthe image sensor 100 and processes the received data by, for example,performing processing related to an image processing method according tothe present embodiment. As described earlier, the image processingapparatus 200 has a role of a receiving apparatus in the imageprocessing system 1000. An example of a configuration related toprocessing of data transmitted from the image sensor 100 (aconfiguration for fulfilling the role as a receiving apparatus) will bedescribed later.

For example, the image processing apparatus 200 is constituted by one ortwo or more processors constituted by an arithmetic circuit such as anMPU (Micro Processing Unit), various processing circuits, and the like.The image processing apparatus 200 operates using power supplied from aninternal power supply (not illustrated) that constitutes the imageprocessing system 1000 such as a battery or power supplied from anexternal power supply of the image processing system 1000.

The image processing apparatus 200 processes image data acquired fromeach of the plurality of image sensors 100 by performing processingrelated to the image processing method according to the presentembodiment.

In the image processing system 1000, the image sensor 100 transmitsregion image data by a transmission system according to a transmissionmethod to be described later. The image processing apparatus 200associatively processes region image data acquired from each of theplurality of image sensors 100 for each region being set with respect toan image.

More specifically for example, the image processing apparatus 200combines images indicated by the region image data acquired from each ofthe plurality of image sensors 100 for each of the regions.

In doing so, the image processing apparatus 200 combines the imagesindicated by the region image data of an object to be combined byaligning relative positions of the images. For example, the imageprocessing apparatus 200 aligns relative positions of images indicatedby region image data based on information related to an angle of viewacquired from each of the image sensors 100 having transmitted theregion image data (or information related to an angle of view acquiredfrom the image sensor module described earlier: hereinafter, a similardescription will apply). In addition, the image processing apparatus 200may align relative positions of images indicated by region image data byperforming arbitrary object detection processing with respect to each ofthe images indicated by region image data and detecting a correspondingobject.

The processing for associating region image data acquired from each ofthe plurality of image sensors 100 with each region is not limited tothe example described above.

For example, the image processing apparatus 200 may combine the imagesindicated by region image data by matching signal levels. The imageprocessing apparatus 200 realizes a combination of matched signal levelsby, for example, “obtaining a correction gain for correcting respectivesensitivity ratios of the image sensors 100 having transmitted regionimage data based on information (to be described later) which isacquired from each of the plurality of image sensors 100 and which isrelated to imaging by the image sensors 100”. In this case, an exampleof sensitivity of the image sensor 100 is a photoelectric conversionratio of an image sensor device included in the image sensor 100.

Processing in the image processing apparatus 200 is not limited to theexample described above.

For example, the image processing apparatus 200 can perform arbitraryprocessing that can be performed with respect to image data such as RGBprocessing, YC processing, and Gamma processing.

In addition, the image processing apparatus 200 performs various typesof processing such as processing related to control of recording ofimage data to a recording medium such as the memory 300, processingrelated to control of display of an image on a display screen of thedisplay device 400, and processing for executing arbitrary applicationsoftware. Examples of processing related to control of recording include“processing for transmitting, to a recording medium such as the memory300, control data including a recording command and data to be recordedin the recording medium”. In addition, examples of processing related tocontrol of display include “processing for transmitting, to a displaydevice such as the display device 400, control data including a displaycommand and data to be displayed on a display screen”.

Furthermore, the image processing apparatus 200 may control functions inthe image sensor 100 by, for example, transmitting control informationto the image sensor 100. The image processing apparatus 200 can alsocontrol data to be transmitted from the image sensor 100 by, forexample, transmitting region designation information to the image sensor100.

An example of a configuration of the image processing apparatus 200 willbe described later.

For example, the image processing system 1000 is configured as shown inFIG. 1 . It should be noted that the configuration of the imageprocessing system according to the present embodiment is not limited tothe example shown in FIG. 1.

For example, in a case where an image transmitted from the image sensor100 is stored in a recording medium outside of the image processingsystem, a case where an image transmitted from the image sensor 100 isstored in a memory provided in the image processing apparatus 200, or acase where an image transmitted from the image sensor 100 is notrecorded, the image processing system according to the presentembodiment need not have the memory 300.

In addition, the image processing system according to the presentembodiment can be configured not to have the display device 400 shown inFIG. 1 .

Furthermore, the image processing system according to the presentembodiment may have an arbitrary configuration in accordance withfunctions provided in an electronic device to be described later towhich the image processing system according to the present embodiment isto be applied.

[2] Application Example of Image Processing System According to PresentEmbodiment

While an image processing system has been described above as the presentembodiment, the present embodiment is not limited to this mode. Forexample, the present embodiment can be applied to various electronicdevices including a communication apparatus such as a smartphone, amobile body such as a drone (a device that can be operated by remotecontrol or a device capable of operating autonomously) or an automobile,a computer such as a PC (personal computer), a tablet-type apparatus,and a gaming console.

[3] Transmission Method According to Present Embodiment

Next, a transmission method according to the present embodiment will bedescribed. Hereinafter, a case where the transmission method accordingto the present embodiment is applied to the image sensor 100 will bedescribed.

(1) First Transmission System

FIG. 5 is an explanatory diagram showing an example of regions to be setwith respect to an image. In FIG. 5 , four regions including region 1,region 2, region 3, and region 4 are shown as an example of regions. Itis needless to say that the regions to be set with respect to an imageare not limited to the example shown in FIG. 5 .

For example, the image sensor 100 transmits, in respectively differentpackets, “additional data including region information corresponding toa region set with respect to an image such as regions 1 to 4 shown inFIG. 5 for each region” and “region image data indicating an image foreach row (line) corresponding to the region”. When a position of a pixelis to be indicated by two-dimensional planar coordinates (x, y), a rowin an image refers to y coordinates being the same.

Region information according to the present embodiment refers to data (adata group) for identifying, from a receiving apparatus side, a regionto be set with respect to an image. For example, region informationincludes a part of or all of identification information of a region,information indicating a position of the region, and informationindicating a size of the region.

Information included in region information is not limited to the exampledescribed above. Region information may include arbitrary informationfor identifying, from the receiving apparatus side, a region to be setwith respect to an image. For example, when regions are divided by VCnumbers, a VC number may perform the role of identification informationof a region included in a row. In addition, when regions are divided byVC numbers, a payload length can be substituted for informationindicating a size of a region included in a row.

Examples of identification information of a region include an arbitrarypiece of data that enables a region to be uniquely identified such asdata indicating an ID of a region such as a number added to the region.Hereinafter, identification information of a region may be referred toas an “ROI ID”.

Information indicating a position of a region is data indicating aposition of the region in an image. An example of information indicatinga position of a region is “data indicating an arbitrary position thatenables a region to be uniquely identified by being combined with a sizeof the region indicated by information indicating the size of theregion” such as data indicating an upper left position of a region in animage by two-dimensional planar coordinates (x, y).

Examples of information indicating a size of a region include dataindicating the number of rows of the region (data indicating the numberof pixels in a vertical direction in the region) and data indicating thenumber of columns of the region (data indicating the number of pixels ina horizontal direction in the region). It should be noted that theinformation indicating a size of a region may be data in an arbitraryformat that enables a size of a region to be identified such as dataindicating a rectangular region (for example, data indicating the numberof pixels in the horizontal direction and the number of pixels in thevertical direction in the rectangular region).

Hereinafter, an example of processing according to the firsttransmission system in the image sensor 100 will be described.

The image sensor 100 stores region information in “Embedded Data” of onepacket and causes the packet to be transmitted. In addition, the imagesensor 100 stores region image data in the payload of another packet andcauses the packet to be transmitted per row.

“Embedded Data” refers to data that can be embedded into a packet to betransmitted and corresponds to additional data that is additionallytransmitted by the image sensor 100. Hereinafter, Embedded Data may alsobe referred to as “EBD”.

FIG. 6 is an explanatory diagram showing an example of data to betransmitted by the first transmission system related to a transmissionmethod according to the present embodiment. FIG. 6 represents “anexample in which region information respectively corresponding to region1, region 2, region 3, and region 4 shown in FIG. 5 is stored as“Embedded Data” in the payload of a long packet of MIPI and region imagedata is stored in the payload of the long packet of MIPI shown in FIG. 3to be transmitted per row”.

“FS” shown in FIG. 6 is an FS (Frame Start) packet in the MIPI CSI-2standard, and “FE” shown in FIG. 6 is an FE (Frame End) packet in theMIPI CSI-2 standard (a similar description applies to other drawings).

As described above, “Embedded Data” shown in FIG. 6 is data that can beembedded into a packet to be transmitted. For example, “Embedded Data”can be embedded into the header, the payload, or the footer of a packetto be transmitted. In the example shown in FIG. 6 , region informationis stored in “Embedded Data” of one packet, and the “Embedded Data” inwhich the region information is stored corresponds to additional data.

The information included in additional data according to the presentembodiment is not limited to the example described above. For example,the additional data according to the present embodiment may includeinformation regarding imaging by the image sensor 100. Examples ofinformation regarding imaging by the image sensor 100 include a part ofor all of exposure information indicating an exposure value or the likein an image sensor device, gain information indicating a gain in theimage sensor device, and sensitivity information indicating aphotoelectric conversion ratio in the image sensor device. Each of theexposure value indicated by the exposure information and the gainindicated by the gain information is set to the image sensor device by,for example, control by the image processing apparatus 200 via thecontrol bus B2.

FIG. 7 is an explanatory diagram for explaining an example of EmbeddedData to be transmitted by the first transmission system according to thepresent embodiment. Data of PH and thereafter shown in FIG. 7 is anexample of Embedded Data shown in FIG. 6 .

In Embedded Data, a type of data included in Embedded Data is definedby, for example, “Data Format Code”.

In the example shown in FIG. 7 , each of “1st ROI Info”, “2nd ROI Info”,. . . following “Data Format Code” corresponds to an example of regioninformation. In other words, Embedded Data shown in FIG. 7 is an exampleof additional data including region information.

For example, in region information shown in FIG. 7 , “Value” includesidentification information of a region, information indicating aposition of the region, and information indicating a size of the region.In addition, “Value” may include information regarding imaging by theimage sensor 100. In the region information shown in FIG. 7 , a boundarywith other pieces of region information included in Embedded Data isdefined by for example, “Length”.

FIG. 8 is an explanatory diagram for explaining an example of regioninformation included in the Embedded Data shown in FIG. 7 . “ROI ID”shown in FIG. 8 corresponds to identification information of a regionand “Upper Left Coordinate” shown in FIG. 8 corresponds to informationindicating a position of the region. In addition, “Height” and “Width”shown in FIG. 8 corresponds to information indicating a size of theregion.

It is needless to say that a data configuration example of the regioninformation and data included in Embedded Data are not limited to theexamples shown in FIG. 7 and FIG. 8 .

Once again referring to FIG. 6 , an example of data to be transmitted bythe first transmission system will be described. Each of “1”, “2”, “3”,and “4” shown in FIG. 6 corresponds to region image data of region 1,region image data of region 2, region image data of region 3, and regionimage data of region 4 to be stored in a payload of a packet. While eachpiece of region image data is shown divided in FIG. 6 , the divisionsare simply indicated for convenience's sake and the data stored in thepayload is not divided.

In the first transmission system, region information respectivelycorresponding to region 1, region 2, region 3, and region 4 shown inFIG. 5 is stored in “Embedded Data” of one packet as shown in FIG. 7 tobe transmitted. In addition, in the first transmission system, regionimage data respectively corresponding to region 1, region 2, region 3,and region 4 shown in FIG. 5 is stored in the payload of a long packetof MIPI as shown in FIG. 6 to be transmitted per row.

(2) Second Transmission System

A transmission method that can be applied to the image processing system1000 according to the present embodiment is not limited to atransmission method according to the first transmission system.

For example, the image sensor 100 may store region information andregion image data in the payload of a packet and cause the packet to betransmitted per row.

FIG. 9 is an explanatory diagram showing another example of regions tobe set with respect to an image. In FIG. 9 , four regions includingregion 1, region 2, region 3, and region 4 are shown as an example ofregions.

FIG. 10 is an explanatory diagram showing an example of data to betransmitted by a second transmission system related to a transmissionmethod according to the present embodiment. FIG. 10 represents “anexample in which region information and region image data respectivelycorresponding to region 1, region 2, region 3, and region 4 shown inFIG. 9 are stored in the payload of a long packet of MIPI as shown inFIG. 3 to be transmitted per row.”

“PH” shown in FIG. 10 denotes a packet header of the long packet. Inthis case, the packet header of the long packet according to the secondtransmission system may function as data (change information) indicatingwhether or not information included in the region information haschanged from region information included in a packet to be lasttransmitted. In other words, “PH” shown in FIG. 10 can be considereddata indicating a data type of the long packet.

As an example, when the information included in the region informationhas changed from region information included in a packet to be lasttransmitted, the image sensor 100 sets “0x38” to “PH”. In this case, theimage sensor 100 stores the region information in the payload of thelong packet.

As another example, when the information included in the regioninformation has not changed from region information included in a packetto be last transmitted, the image sensor 100 sets “0x39” to “PH”. Inthis case, the image sensor 100 does not store the region information inthe payload of the long packet. In other words, when the informationincluded in the region information has not changed from regioninformation included in a packet to be last transmitted, the imagesensor 100 does not cause region information to be transmitted.

It is needless to say that the data to be set to “PH” is not limited tothe example described above.

“Info” in FIG. 10 denotes region information stored in a payload. Asshown in FIG. 10 , region information is stored in a head portion of apayload.

Each of “1”, “2”, “3”, and “4” shown in FIG. 10 corresponds to regionimage data of region 1, region image data of region 2, region image dataof region 3, and region image data of region 4 to be stored in apayload. While each piece of region image data is shown divided in FIG.10 , the divisions are simply indicated for convenience's sake and thedata stored in the payload is not divided.

In the second transmission system, region information and region imagedata respectively corresponding to region 1, region 2, region 3, andregion 4 shown in FIG. 9 are stored in the payload of a long packet ofMIPI as shown in, for example, FIG. 10 to be transmitted per row.

Therefore, when the second transmission system is used, the image sensor100 is capable of transmitting a shape of an arbitrary region set to theimage.

[4] Configuration Example of Image Sensor and Image Processing ApparatusConstituting Image Processing System According to Present Embodiment

Next, an example of a configuration of the image sensor 100 capable ofperforming processing according to the transmission method describedabove and a configuration of the image processing apparatus 200 capableof performing processing according to the image processing methoddescribed above will be explained.

[4-1] Configuration of Image Sensor 100

FIG. 11 is a block diagram showing an example of a configuration of theimage sensor 100 according to the present embodiment. For example, theimage sensor 100 includes a photoelectric conversion unit 102, a signalprocessing unit 104, a communicating unit 106, and a control unit 108.The image sensor 100 operates using power supplied from an internalpower supply (not illustrated) that constitutes the image processingsystem 1000 such as a battery or power supplied from an external powersupply of the image processing system 1000.

The photoelectric conversion unit 102 is constituted by a lens/imagingelement 150 and the signal processing unit 104 is constituted by asignal processing circuit 152. The lens/imaging element 150 and thesignal processing circuit 152 function as an image sensor device in theimage sensor 100. In the image processing system 1000, all of the imagesensors 100 may include image sensor devices of a same type or a part ofthe image sensors 100 may include image sensor devices of a differenttype. An example of image sensors 100 including image sensor devices ofa different type is an image sensor 100 including an image sensor devicethat images a color image and an image sensor 100 including an imagesensor device that images a black and white image.

The communicating unit 106 is constituted by a communication circuit 154and the control unit 108 is constituted by a processor 156. Operationsof each of the lens/imaging element 150, the signal processing circuit152, and the communication circuit 154 are controlled by the processor156.

It should be noted that the function blocks of the image sensor 100shown in FIG. 11 have been created by dividing functions included in theimage sensor 100 for convenience's sake and are not limited to theexample shown in FIG. 11 . For example, the signal processing unit 104and the control unit 108 shown in FIG. 11 can also be considered asingle processing unit.

The lens/imaging element 150 is constituted by, for example, a lens ofan optical system and an image sensor using a plurality of imagingelements such as a CMOS (Complementary Metal Oxide Semiconductor) or aCCD (Charge Coupled Device). In the lens/imaging element 150, as lighthaving passed through the lens of the optical system isphotoelectrically converted by the imaging elements of the image sensor,an analog signal indicating a captured image is obtained.

For example, the signal processing circuit 152 includes an AGC(Automatic Gain Control) circuit and an ADC (Analog to DigitalConverter) and converts an analog signal transmitted from thelens/imaging element 150 into a digital signal (image data). Inaddition, the signal processing circuit 152 includes an amplifier andamplifies the digital signal with a prescribed gain.

Furthermore, the signal processing circuit 152 may perform processingrelated to setting a region with respect to the image and transmitregion designation information to the communication circuit 154. As willbe described later, processing related to setting of a region withrespect to the image in the image sensor 100 may be performed by theprocessor 156. In addition, as described earlier, in the imageprocessing system 1000, processing related to setting of a region withrespect to an image may be performed by an external apparatus such asthe image processing apparatus 200.

Furthermore, the signal processing circuit 152 may transmit variouspieces of data such as exposure information and gain information to thecommunication circuit 154. Transmission of various pieces of data suchas exposure information and gain information to the communicationcircuit 154 in the image sensor 100 may be performed by the processor156.

The communication circuit 154 is a circuit related to a datatransmission function by the transmission method according to thepresent embodiment and an example of the communication circuit 154 is anIC (Integrated Circuit) chip in which circuits related to thetransmission function are integrated. The communication circuit 154processes image data transmitted from the signal processing circuit 152and transmits date corresponding to a generated image.

Data corresponding to an image is image data transmitted from the signalprocessing circuit 152 (in other words, data indicating an entire image)or region information and region image data.

The processor 156 controls operations of each of the lens/imagingelement 150, the signal processing circuit 152, and the communicationcircuit 154 based on, for example, a control signal transmitted from theimage processing apparatus 200 via the control bus B2. Alternativelywhen the image sensor 100 provided with the processor 156 and anotherimage sensor 100 are capable of directly communicating with each other,the processor 156 can perform processing based on a control signaltransmitted from the other image sensor 100 via an arbitrarytransmission path.

Examples of control of the lens/imaging element 150 by the processor 156include control of imaging such as control of an exposure time. Examplesof control of the signal processing circuit 152 by the processor 156include control of signal processing such as control of a gain. Examplesof control of the communication circuit 154 by the processor 156 includecontrol of communication such as “control of switching betweentransmitting region image data and transmitting data indicating anentire image” and various types of control when transmitting regionimage data (for example, control of transmission of region informationand control of transmission of information related to imaging).

The image sensor 100 performs processing related to the transmissionmethod described above by for example, the configuration shown in FIG.11 . It is needless to say that the configuration of the image sensor100 is not limited to the example shown in FIG. 11 .

[4-2] Configuration of Image Processing Apparatus 200

FIG. 12 is a block diagram showing an example of a configuration of theimage processing apparatus 200 according to the present embodiment. FIG.12 shows an example of a configuration of the image processing apparatus200 that constitutes the image processing system 1000 shown in FIG. 1or, in other words, an example of a configuration which communicateswith each of two image sensors 100, namely the image sensors 100A and100B.

For example, the image processing apparatus 200 includes a communicatingunit 202 and a processing unit 204. The image processing apparatus 200operates using power supplied from an internal power supply (notillustrated) that constitutes the image processing system 1000 such as abattery or power supplied from an external power supply of the imageprocessing system 1000.

The communicating unit 202 has a function of communicating with each ofthe plurality of image sensors 100. For example, the communicating unit202 is constituted by communication circuits 250A and 250B whichrespectively correspond to the image sensor 100 that is a communicationobject. Hereinafter, one communication circuit among the communicationcircuits 250A and 250B that constitute the communicating unit 202 willbe referred to as a “communication circuit 250”.

In addition, the communicating unit 202 may be capable of switchingamong image sensors 100 that are communication objects. Using the imageprocessing system 1000 shown in FIG. 1 as an example, switching of theimage sensors 100 that are communication objects in the communicatingunit 202 include switching among “communicate with only the image sensor100A”, “communicate with only the image sensor 100B”, and “communicatewith both the image sensor 100A and the image sensor 100B”. Theswitching of the image sensors 100 that are communication objects in thecommunicating unit 202 is realized as, for example, operations of thecommunication circuits 250A and 250B are controlled by a processor 252.The processor 252 performs switching of the image sensors 100 that arecommunication objects by threshold processing based on a detected valueof a sensor capable of detecting brightness such as an illuminancesensor (which may either be a sensor outside of the image processingapparatus 200 or a sensor included in the image processing apparatus200). As an example, the processor 252 causes communication with boththe image sensor 100A and the image sensor 100B to be performed when thedetected value is equal to or smaller than a set threshold (or when thedetected value is smaller than the threshold). As another example, theprocessor 252 causes communication with one of the image sensor 100A andthe image sensor 100B to be performed when the detected value is largerthan the threshold (or when the detected value is equal to or largerthan the threshold). Since a data processing amount in the imageprocessing apparatus 200 can be further reduced by having the processor252 switch the image sensors 100 that are communication objects, areduction in power consumption can be achieved.

In addition, when “communication with only the image sensor 100A” or“communication with only the image sensor 100B” is performed, theprocessor 252 may suspend operations of the image sensor 100 that doesnot perform communication. For example, the processor 252 performsswitching of the image sensors 100 that are communication objects andsuspension of operations of the image sensors 100 by thresholdprocessing based on a detected value of a sensor capable of detectingbrightness such as an illuminance sensor. By having the processor 252suspend operations of the image sensors 100, a reduction in powerconsumption can be achieved in the image processing system 1000.

The processing unit 204 processes data received by the communicatingunit 202. For example, the processing unit 204 performs processingrelated to the image processing method according to the presentembodiment and, based on region information, associatively processesregion image data acquired from each of the plurality of image sensors100 for each region. Alternatively, the processing unit 204 can alsoprocess data indicating an entire image.

The processing unit 204 is constituted by the processor 252 and an imageprocessing circuit 254. Operations of each of the communication circuits250A and 250B and the image processing circuit 254 are controlled by theprocessor 252. In other words, the processing unit 204 may perform arole of a control unit in the image processing apparatus 200.

In addition, the processor 252 constituting the processing unit 204performs a role of controlling operations of each of the image sensors100 constituting the image processing system 1000. The processor 252controls operations of each of the image sensors 100 by transmitting acontrol signal to the image sensors 100 via the control bus B2.

It should be noted that the function blocks of the image processingapparatus 200 shown in FIG. 12 have been created by dividing functionsincluded in the image processing apparatus 200 for convenience's sakeand are not limited to the example shown in FIG. 12 . For example, theprocessing unit 204 shown in FIG. 12 can also be divided into a controlunit constituted by the processor 252 and an image processing unitconstituted by the image processing circuit 254.

The communication circuit 250A is, for example, a communication circuitthat communicates with the image sensor 100A. The communication circuit250A receives data (for example, the packet shown in FIG. 6 or FIG. 10 )having been transmitted by the transmission method according to thepresent embodiment from the image sensor 100A. The communication circuit250A may have a function of transmitting data to the image sensor 100Avia, for example, an arbitrary transmission path between thecommunication circuit 250A and the image sensor 100A.

The communication circuit 250B is, for example, a communication circuitthat communicates with the image sensor 100B. The communication circuit250B receives data (for example, the packet shown in FIG. 6 or FIG. 10 )having been transmitted by the transmission method according to thepresent embodiment from the image sensor 100B. The communication circuit250B may have a function of transmitting data to the image sensor 100Bvia, for example, an arbitrary transmission path between thecommunication circuit 250B and the image sensor 100B.

The communication circuits 250A and 250B transmit data included inEmbedded Data such as region information and information regardingimaging by the image sensors 100 among the received data to theprocessor 252. FIG. 12 shows “an example in which region information istransmitted to the processor 252 from each of the communication circuits250A and 250B”. The communication circuits 250A and 250B may transmitEmbedded Data among the received data to the processor 252. WhenEmbedded Data is transmitted to the processor 252, the data included inEmbedded Data such as region information and information regardingimaging by the image sensors 100 is retrieved by the processor 252 fromEmbedded Data. In addition, the communication circuits 250A and 250Btransmit data other than the Embedded Data included in the payload amongthe received data to the image processing circuit 254.

The communication circuits 250A and 250B separate header datacorresponding to a header portion and payload data corresponding to apayload portion from the received data. The communication circuits 250Aand 250B separate header data from the received data according to, forexample, a rule defined in advance by a standard or the like. Inaddition, the communication circuits 250A and 250B may separate payloaddata from the received data or separate payload data from the receiveddata based on contents indicated by the header data according to, forexample, a rule defined in advance by a standard or the like.Furthermore, the communication circuits 250A and 250B transmit dataincluded in Embedded Data (or Embedded Data) among the separated data tothe processor 252 and transmit data other than the Embedded Data amongthe payload data to the image processing circuit 254.

FIG. 13 is a block diagram showing an example of a functionalconfiguration of the communication circuit 250 included in the imageprocessing apparatus 200 according to the present embodiment. Forexample, the communication circuit 250 includes a header separating unit260, a header interpreting unit 262, and a payload separating unit 264.

The header separating unit 260 separates header data corresponding to aheader portion and payload data corresponding to a payload portion fromthe received data. The header separating unit 260 separates header datafrom the received data according to, for example, a rule defined inadvance by a standard or the like. In addition, the header separatingunit 260 may separate payload data from the received data or separatepayload data from the data received based on a result of processing bythe header interpreting unit 262 according to, for example, a ruledefined in advance by a standard or the like.

The header interpreting unit 262 interprets contents indicated by theheader data.

As an example, the header interpreting unit 262 interprets whether ornot payload data is “Embedded Data”. The header interpreting unit 262interprets whether or not payload data is “Embedded Data” based on, forexample, a DT value recorded in the header portion. As another example,the header interpreting unit 262 may identify a position of payload dataand transmit the identified position to the header separating unit 260.

The payload separating unit 264 processes the payload data based on aresult of interpretation by the header interpreting unit 262.

As an example, when the header interpreting unit 262 interprets that thepayload data is “Embedded Data”, the payload separating unit 264separates data included in Embedded Data such as region information andinformation regarding imaging by the image sensors 100 from the payloaddata. In addition, the payload separating unit 264 transmits the dataincluded in Embedded Data such as region information and informationregarding imaging by the image sensors 100 to the processing unit 204(more specifically, the processor 252 constituting the processing unit204). FIG. 13 shows “an example in which region information istransmitted to the processing unit 204 from the payload separating unit264”.

As another example, when the header interpreting unit 262 does notinterpret that the payload data is “Embedded Data”, the payloadseparating unit 264 separates image data (data indicating an entireimage or region image data) from the payload data. The payloadseparating unit 264 separates region image data from the payload databased on, for example, region information retrieved from Embedded Data.In addition, the payload separating unit 264 transmits the image data tothe processing unit 204 (more specifically the image processing circuit254 constituting the processing unit 204).

Due to the communication circuit 250 having a functional configurationshown in, for example, FIG. 13 , the communication circuit 250 receivesdata having been transmitted from the image sensor 100 by thetransmission method according to the present embodiment and transmitsthe received data to the processing unit 204. It should be noted thatthe function blocks of the communication circuit 250 shown in FIG. 13have been created by dividing functions included in the communicationcircuit 250 for convenience's sake and are not limited to the exampleshown in FIG. 13 . In addition, as described above, the communicationcircuit 250 may be configured to transmit Embedded Data among thereceived data to the processor 252.

Once again referring to FIG. 12 , an example of a configuration of theimage processing apparatus 200 will be described. The processor 252controls operations of each of the communication circuits 250A and 250Band the image processing circuit 254. In addition, the processor 252 mayperform various kinds of processing such as processing for executingarbitrary application software.

Examples of control of the communication circuits 250A and 250B by theprocessor 252 include on/off control of a communication function. Forexample, by controlling on/off of the communication function of each ofthe communication circuits 250A and 250B as described above, switchingamong image sensors 100 that are communication objects is realized.

Control of the image processing circuit 254 by the processor 252 includecontrol of processing related to the image processing method accordingto the present embodiment that is performed by the image processingcircuit 254. The processor 252 performs control of the image processingcircuit 254 using, for example, data included in Embedded Data such asregion information and information regarding imaging by the imagesensors 100 that is transmitted from the communication circuits 250A and250B. In addition, when Embedded Data is transmitted from thecommunication circuits 250A and 250B, the processor 252 performs controlof the image processing circuit 254 by for example, retrieving regioninformation or the like from the Embedded Data.

As an example, the processor 252 transmits, to the image processingcircuit 254, correction control information that indicates a correctionvalue for aligning a relative position of an image indicated by theregion image data. The correction value for aligning a relative positionof an image indicated by the region image data is set based on, forexample, region information included in Embedded Data transmitted fromeach of the communication circuits 250A and 250B and informationregarding an angle of view acquired from each of the image sensors 100.Alternatively, the correction value for aligning a relative position ofan image indicated by the region image data may be set based on, forexample, region information included in Embedded Data transmitted fromeach of the communication circuits 250A and 250B and a result ofperforming arbitrary object detection processing with respect to eachimage indicated by the region image data.

As another example, the processor 252 transmits, to the image processingcircuit 254, correction control information that indicates a correctiongain for correcting a sensitivity ratio of each of the image sensors 100having transmitted region image data.

The correction gain is set by calculating a correction gain “Gl2” thatsatisfies mathematical expression 1 below based on, for example,information regarding an angle of view that is acquired from each of theimage sensors 100. It should be noted that the calculation of thecorrection gain based on mathematical expression 1 below is acalculation example in a case where the image processing apparatus 200controls the image sensors 100 so that respective exposure times are thesame. In other words, a calculation method of the correction gainaccording to the present embodiment is not limited to using mathematicalexpression 1 below.

G2·G2l=A1·G1/A2   (Mathematical expression 1)

In this case, “G1” in mathematical expression 1 presented above denotesa gain in the image sensor device included in the image sensor 100A and“G2” in mathematical expression 1 presented above denotes a gain in theimage sensor device included in the image sensor 100B. In addition, “A1”in mathematical expression 1 presented above denotes a photoelectricconversion ratio in the image sensor device included in the image sensor100A and “A2” in mathematical expression 1 presented above denotes aphotoelectric conversion ratio in the image sensor device included inthe image sensor 100B. In other words, a correction gain for correctinga signal level of an image indicated by region image data acquired fromthe image sensor 100B is calculated using mathematical expression 1presented above.

The image processing circuit 254 processes data transmitted from each ofthe communication circuits 250A and 250B. For example, in the imageprocessing circuit 254, the processing unit 204 performs processingrelated to the image processing method according to the presentembodiment and, based on region information, associatively processesregion image data acquired from each of the image sensors 100A and 100Bfor each region.

For example, the image processing circuit 254 matches a signal level ofan image indicated by region image data using, for example, correctioncontrol information transmitted from the processor 252. After matchingthe signal level, the image processing circuit 254 aligns a relativeposition of an image indicated by the region image data using thecorrection control information transmitted from the processor 252. Inaddition, the image processing circuit 254 combines an image indicatedby area image data for each region. It should be noted that the imageprocessing circuit 254 is capable of combining an image indicated byregion image data without having to match a signal level of the imageindicated by region image data and is capable of combining an imageindicated by region image data without having to align a relativeposition of the image indicated by region image data.

Alternatively, the image processing circuit 254 can also process dataindicating an entire image acquired from each of the image sensors 100Aand 100B.

In addition, processing in the image processing circuit 254 is notlimited to the example described above. For example, the imageprocessing circuit 254 may perform one of or both of processing relatedto control of recording of image data to a recording medium such as thememory 300 and processing related to control of display of an image on adisplay screen of the display device 400.

FIG. 14 is a block diagram showing an example of a functionalconfiguration of the image processing circuit 254 included in the imageprocessing apparatus 200 according to the present embodiment. Forexample, the image processing circuit 254 has first image processingunits 270A and 270B, a relative sensitivity difference correctionprocessing unit 272, a relative position correction processing unit 274,a combination processing unit 276, and a second image processing unit278. Apart of or all of processing in each unit may be performed byhardware or performed by having software (a computer program) executedon hardware.

Hereinafter, an example of the functional configuration of the imageprocessing circuit 254 will be described using a case where the imageprocessing circuit 254 processes region image data as an example.

The first image processing unit 270A performs prescribed imageprocessing with respect to data transmitted from the communicationcircuit 250A. The first image processing unit 270B performs prescribedimage processing with respect to data transmitted from the communicationcircuit 250B. Examples of the prescribed image processing performed byeach of the first image processing units 270A and 270B include varioustypes of processing related to a RAW phenomenon or the like.

The relative sensitivity difference correction processing unit 272matches a signal level of an image indicated by region image datatransmitted from the first image processing unit 270B to a signal levelof an image indicated by region image data having been processed by thefirst image processing unit 270A. The relative sensitivity differencecorrection processing unit 272 corrects a gain of the region image datatransmitted from the first image processing unit 270B using, forexample, a correction gain indicated by correction control informationtransmitted from the processor 252.

While FIG. 14 shows an example in which the gain of region image datatransmitted from the first image processing unit 270B is corrected, theimage processing circuit 254 may have a functional configuration forcorrecting a gain of the region image data transmitted from the firstimage processing unit 270A.

The relative position correction processing unit 274 aligns a relativeposition of an image indicated by region image data transmitted from therelative sensitivity difference correction processing unit 272 to animage indicated by region image data having been processed by the firstimage processing unit 270A. The relative position correction processingunit 274 corrects a relative position of an image indicated by theregion image data transmitted from the relative sensitivity differencecorrection processing unit 272 using, for example, a correction gain foraligning a relative position indicated by correction control informationtransmitted from the processor 252.

The combination processing unit 276 combines, for each region, an imageindicated by region image data processed by the first image processingunit 270A and an image indicated by region image data transmitted fromthe relative position correction processing unit 274. The combinationprocessing unit 276 combines images indicated by pieces of region imagedata by arbitrary processing that enables images to be combined such asalpha blending.

The second image processing unit 278 performs prescribed imageprocessing with respect to a combined image transmitted from thecombination processing unit 276. Examples of the prescribed imageprocessing performed by the second image processing unit 278 includesarbitrary processing that can be performed with respect to image datasuch as Gamma processing.

Due to the image processing circuit 254 having a functionalconfiguration shown in, for example, FIG. 14 , the image processingcircuit 254 performs processing related to the image processing methodaccording to the present embodiment. It should be noted that thefunction blocks of the image processing circuit 254 shown in FIG. 14have been created by dividing functions included in the image processingcircuit 254 for convenience's sake and are not limited to the exampleshown in FIG. 14 .

The image processing apparatus 200 performs processing related to theimage processing method described above by, for example, theconfiguration shown in FIG. 12 to FIG. 14 . It is needless to say thatthe configuration of the image processing apparatus 200 is not limitedto the example shown in FIG. 12 to FIG. 14 .

[5] Example of Processing in Image Processing System According toPresent Embodiment

Next, an example of processing in the image processing system 1000 willbe described.

[5-1] Processing Related to Initialization

FIG. 15 is an explanatory diagram for explaining an example ofprocessing in the image processing system 1000 according to the presentembodiment and shows an example of processing related to initialization.The processing shown in FIG. 15 is performed when, for example, theimage processing system 1000 is activated or when a prescribed operationis performed by a user of the image processing system 1000 or the like.

The image processing apparatus 200 transmits a setting request to haveeach of the image sensors 100A and 100B set a drive parameter and anacquisition request to have each of the image sensors 100A and 100Btransmit information regarding an angle of view via, for example, thecontrol bus B2 (S100). For example, the setting request of a driveparameter includes various setting values such as an exposure value, anexposure time, and a gain and a setting command. For example, theacquisition request includes a transmission command of informationrelated to an angle of view.

Each of the image sensors 100A and 100B having received the settingrequest and the acquisition request transmitted in step S100 sets adrive parameter based on the setting request and transmits informationrelated to an angle of view based on the acquisition request (S102,S104).

The image processing apparatus 200 having transmitted the settingrequest in step S100 calculates a correction gain based on varioussetting values included in the setting request and configures settingsfor performing correction according to the calculated correction gain(S106).

The image processing apparatus 200 having received information regardingthe angle of view transmitted in steps S102 and S104 obtains acorrection value for aligning relative positions based on theinformation regarding the angle of view (S108) and configures settingsfor performing correction according to the correction value (S110).

In the image processing system 1000, for example, processing shown inFIG. 15 is performed as processing related to initialization. It isneedless to say that the example of processing related to initializationis not limited to the example shown in FIG. 15 .

[5-2] Processing During Operation

FIG. 16 is an explanatory diagram for explaining an example ofprocessing in the image processing system 1000 according to the presentembodiment and shows an example of processing during operation. FIG. 16shows an example in which processing by the image sensor 100B isperformed with imaging by the image sensor 100A as a reference. In otherwords, the image processing system 1000 can perform coordinated imagingby having one image sensor 100 function as a master image sensor andhaving another image sensor 100 function as a slave image sensor.

The image sensor 100A starts imaging upon acquiring a frame starttrigger (hereinafter, sometimes referred to as a “V Start Trigger”)(S200).

The image sensor 100A sets a cutout position to be cut out from acaptured image (S202) and transmits information indicating the setcutout position to the image sensor 100B and the image processingapparatus 200 (S204). Setting of the cutout position in the image sensor100A corresponds to setting a region with respect to the captured image.In other words, for example, the information indicating the cutoutposition corresponds to region information.

The image sensor 100A transmits information indicating the cutoutposition to the image processing apparatus 200 via, for example, thedata bus B1. In addition, the image sensor 100A transmits information(region information: hereinafter, a similar description will apply)indicating the cutout position to the image sensor 100B via, forexample, the image processing apparatus 200. In a case of aconfiguration in which the image sensor 100A and the image sensor 100Bare capable of communicating with each other by inter-processorcommunication or the like, the image sensor 100A may transmitinformation indicating the cutout position to the image sensor 100B bydirect communication.

The image sensor 100B having received information indicating a cutoutposition transmitted from the image sensor 100A in step S204 sets acutout position to be cut out from a captured image based on theinformation indicating the cutout position (S206).

The image processing apparatus 200 having received informationindicating a cutout position transmitted from the image sensor 100A instep S204 identifies the number of pixels included in the set region andtwo-dimensional planar coordinates of the pixels based on theinformation indicating the cutout position (S208) and sets coordinatesand a size of a region to be used in processing (S210).

The image sensor 100A having transmitted the information indicating acutout position in step S204 transmits information regarding imaging inthe image sensor 100A to the image sensor 100B and the image processingapparatus 200 (S212). As described earlier, information regardingimaging includes exposure information and gain information. The imagesensor 100A transmits information regarding imaging to the image sensor100B and the image processing apparatus 200 in a similar manner to, forexample, the transmission of information indicating a cutout position instep S204.

The image sensor 100B having received the information regarding imagingtransmitted from the image sensor 100A in step S212 performs gaincontrol and exposure control based on the received information regardingimaging (S214). In addition, the image sensor 100B transmits informationregarding imaging in the image sensor 100B to the image processingapparatus 200.

The image processing apparatus 200 having received the informationregarding imaging transmitted from the image sensor 100A in step S212and the information regarding imaging transmitted from the image sensor100B calculates, for example, a correction gain and configures settingsfor performing correction according to the calculated correction gain(S216). In addition, the image processing apparatus 200 startsprocessing with respect to image data transmitted from each of the imagesensors 100A and 100B (S218).

FIG. 17 is an explanatory diagram for explaining an example ofprocessing in the image processing system 1000 according to the presentembodiment and shows a timing chart corresponding to the processingshown in FIG. 16 . In other words, FIG. 17 shows “an example in whichthe image sensor 100A functions as a master image sensor and the imagesensor 100B functions as a slave image sensor” in a similar manner toFIG. 16 .

As shown in FIG. 17 , in the image processing system 1000, the imagesensor 100A and the image sensor 100B cooperate with each other toperform imaging in accordance with a notification of settings in theimage sensor 100A. In addition, in the image processing system 1000, theimage processing apparatus 200 associatively processes region image dataacquired from each of the image sensors 100A and 100B for each regionbased on a notification of settings in the image sensor 100A. Therefore,coordinated operations of the image sensor 100A, the image sensor 100B,and the image processing apparatus 200 are realized in the imageprocessing system 1000.

In the image processing system 1000, for example, processing shown inFIG. 16 and FIG. 17 is performed as processing during operation. It isneedless to say that the example of processing during operation is notlimited to the example shown in FIG. 16 and FIG. 17 .

[6] Example of Advantageous Effect Produced by Using Image ProcessingSystem According to Present Embodiment

For example, using the image processing system according to the presentembodiment produces the advantageous effects described below. It isneedless to say that the advantageous effects produced by using theimage processing system according to present embodiment is not limitedto the examples described below.

-   -   In the image processing system according to the present        embodiment, imaging of a region set with respect to a captured        image can be performed by cooperation among a plurality of the        image sensors 100.    -   In the image processing system according to the present        embodiment, since the plurality of image sensors and the image        processing apparatus operate in cooperation with each other, for        example, the apparatuses can operate while sharing various kinds        of information such as information on an exposure time, a drive        frequency a gain value, a difference in relative angles of view        between image sensor devices, and subject distance.    -   Since the image processing apparatus is capable of combining        images indicated by region image data by matching signal levels        of the images, the image processing system according to the        present embodiment is capable of enhancing sensitivity of images        to be associatively processed.    -   The image processing apparatus is capable of switching among        image sensors that are communication objects and suspending        operations of a part of the image sensors in conjunction with        such switching. Therefore, the image processing system according        to present embodiment is capable of achieving a reduction in        power consumption.

Program According to Present Embodiment

By having a processor or an image processing circuit in a computerexecute a program that causes the computer to function as the imageprocessing apparatus according to the present embodiment (for example, aprogram that causes the computer to execute processing related to theimage processing method according to the present embodiment), imagesrespectively obtained from a plurality of image sensors can beassociatively processed.

In addition, by having a processor or an image processing circuit in acomputer execute a program that causes the computer to function as theimage processing apparatus according to the present embodiment,advantageous effects to be produced through the use of the imageprocessing method according to the present embodiment can be produced.

While a preferred embodiment of the present disclosure has beendescribed in detail with reference to the accompanying drawings, thetechnical scope of the present disclosure is not limited thereto. Itwill be obvious to a person with ordinary skill in the art to which thetechnical field of the present disclosure pertains that variousmodifications and changes can be arrived at without departing from thescope of the technical ideas as set forth in the appended claims and, assuch, it is to be understood that such modifications and changes are tobe naturally covered in the technical scope of the present disclosure.

For example, while a program (a computer program) that causes a computerto function as the image processing apparatus according to the presentembodiment is provided in the description given above, the presentembodiment can also provide a recording medium storing the programdescribed above.

The configuration described above represents an example of the presentembodiment and naturally falls within the technical scope of the presentdisclosure.

Furthermore, the advantageous effects described in the presentspecification are merely descriptive or exemplary and not restrictive.In other words, the technique according to the present disclosure canproduce, in addition to or in place of the advantageous effectsdescribed above, other advantageous effects that will obviously occur tothose skilled in the art from the description of the presentspecification.

The following configurations are also covered in the technical scope ofthe present disclosure.

(1)

An image processing apparatus, including:

-   -   a communicating unit capable of communicating with each of a        plurality of image sensors configured to transmit, in        respectively different packets, additional data including region        information corresponding to a region set with respect to a        captured image for each region and region image data indicating        an image for each row corresponding to the region; and    -   a processing unit configured to process, in association with        each region, the region image data acquired from each of the        plurality of image sensors based on the region information        included in the additional data acquired from each of the        plurality of image sensors, wherein    -   the region information includes a part of or all of        identification information of the region, information indicating        a position of the region, and information indicating a size of        the region.

(2)

The image processing apparatus according to (1), wherein the processingunit is configured to combine images indicated by the region image dataacquired from each of the plurality of image sensors for each region.

(3)

The image processing apparatus according to (2), wherein the processingunit is configured to combine images indicated by the region image dataof an object to be combined by aligning relative positions of theimages.

(4)

The image processing apparatus according to (2) or (3), wherein theadditional data includes information regarding imaging in the imagesensor, and the processing unit is configured to combine imagesindicated by the region image data of an object to be combined bymatching signal levels of the images based on information regarding theimaging having been acquired from each of the plurality of imagesensors.

(5)

The image processing apparatus according to any one of (1) to (4),wherein the communicating unit is configured to be capable of switchingamong image sensors that are communication objects.

(6)

The image processing apparatus according to any one of (1) to (5),wherein the packet is a long packet of MIPI (Mobile Industry ProcessorInterface Alliance).

(7)

An image processing system, including:

-   -   a plurality of image sensors configured to transmit, in        respectively different packets, additional data including region        information corresponding to a region set with respect to a        captured image for each region and region image data indicating        an image for each row corresponding to the region; and    -   an image processing apparatus, wherein    -   the image processing apparatus includes:    -   a communicating unit capable of communicating with each of the        plurality of image sensors; and    -   a processing unit configured to process, in association with        each region, the region image data acquired from each of the        plurality of image sensors based on the region information        included in the additional data acquired from each of the        plurality of image sensors, and    -   the region information includes a part of or all of        identification information of the region, information indicating        a position of the region, and information indicating a size of        the region.

REFERENCE SIGNS LIST

-   -   100, 100A, 100B Image sensor    -   102 Photoelectric conversion unit    -   104 Signal processing unit    -   106, 202 Communicating unit    -   150 Lens/imaging element    -   152 Signal processing circuit    -   154, 250, 250A, 250B Communication circuit    -   156, 252 Processor    -   200 Image processing apparatus    -   204 Processing unit    -   254 Image processing circuit    -   260 Header separating unit    -   262 Header interpreting unit    -   264 Payload separating unit    -   270A, 270B First image processing unit    -   272 Relative sensitivity difference correction processing unit    -   274 Relative position correction processing unit    -   276 Combination processing unit    -   278 Second image processing unit    -   300 Memory    -   400 Display device    -   1000 Image processing system    -   B1 Data bus    -   B2 Control bus

1. An image processing apparatus, comprising: a communicating unitcapable of communicating with each of a plurality of image sensorsconfigured to transmit, in respectively different packets, additionaldata including region information corresponding to a region set withrespect to a captured image for each region and region image dataindicating an image for each row corresponding to the region; and aprocessing unit configured to process, in association with each region,the region image data acquired from each of the plurality of imagesensors based on the region information included in the additional dataacquired from each of the plurality of image sensors, wherein the regioninformation includes a part of or all of identification information ofthe region, information indicating a position of the region, andinformation indicating a size of the region.
 2. The image processingapparatus according to claim 1, wherein the processing unit isconfigured to combine images indicated by the region image data acquiredfrom each of the plurality of image sensors for each region.
 3. Theimage processing apparatus according to claim 2, wherein the processingunit is configured to combine images indicated by the region image dataof an object to be combined by aligning relative positions of theimages.
 4. The image processing apparatus according to claim 2, whereinthe additional data includes information regarding imaging in the imagesensor, and the processing unit is configured to combine imagesindicated by the region image data of an object to be combined bymatching signal levels of the images based on information regarding theimaging having been acquired from each of the plurality of imagesensors.
 5. The image processing apparatus according to claim 1, whereinthe communicating unit is configured to be capable of switching amongimage sensors that are communication objects.
 6. The image processingapparatus according to claim 1, wherein the packet is a long packet ofMIPI (Mobile Industry Processor Interface Alliance).
 7. An imageprocessing system, comprising: a plurality of image sensors configuredto transmit, in respectively different packets, additional dataincluding region information corresponding to a region set with respectto a captured image for each region and region image data indicating animage for each row corresponding to the region; and an image processingapparatus, wherein the image processing apparatus includes: acommunicating unit capable of communicating with each of the pluralityof image sensors; and a processing unit configured to process, inassociation with each region, the region image data acquired from eachof the plurality of image sensors based on the region informationincluded in the additional data acquired from each of the plurality ofimage sensors, and the region information includes a part of or all ofidentification information of the region, information indicating aposition of the region, and information indicating a size of the region.